Skip to content

Phase 6: redirect zarr bytes for non-browser clients - #1202

Merged
neuromechanist merged 9 commits into
feature/issue-1181-epic-zarr-servingfrom
feature/issue-1061-phase6-redirect-contract
Sep 2, 2026
Merged

neuromechanist merged 9 commits into
feature/issue-1181-epic-zarr-servingfrom
feature/issue-1061-phase6-redirect-contract

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Non-browser requests to zarr store objects now redirect straight to the
public S3 object instead of proxying through the Worker, so Cloudflare
never carries the bulk of the byte traffic while every request stays
counted. index.json remains the fully-managed contract entry point.

  • Adds a shared isRedirectCandidate(method, path, origin) predicate in
    backend/src/routes/zarr-data.ts, used identically by serve()'s own
    routing fork and by the zarr sub-app's rate-limit middleware wrapper, so
    routing and rate-limiting can never disagree about which requests reach
    D1/the edge cache/S3 through the Worker.
  • The redirect branch runs before the D1 visibility gate and never touches
    the edge cache or performs an upstream fetch. The bucket's own
    NotResource deny-list (backend/src/services/bucket-policy.ts) is the
    real enforcement point for a private dataset on that path -- a redirect
    that 403s at S3 leaks nothing the proxied 404 does not.
  • The zarr sub-app's rate-limit wrapper exempts redirect candidates from
    the data-ip bucket (logs a counter via console.info instead of
    enforcing), without touching the shared rateLimiter used by the API app.
  • Telemetry: recordAccess fires on the redirect path with
    detail: "chunk-redirect" (the access-metrics blob slots are fixed, so
    this reuses the existing detail slot rather than adding a column) and
    bytes computed from the client's Range header (bytesFromRangeHeader) --
    bounded and suffix ranges get an exact count, open-ended/missing/malformed
    ranges fall back to 0.

Routing table

Request Response
GET /<id>/zarr/index.json Proxied, edge-cached, D1-gated. Unchanged.
GET /<id>/zarr/<store>/... with an allowlisted Origin Proxied with CORS and the phase 1 cache. Unchanged.
GET /<id>/zarr/<store>/... with no Origin, or a non-allowlisted one 302 to the public S3 URL (Location, Cache-Control: public, max-age=3600, Vary: Origin; no Content-Length, no ACAO).
HEAD anything Never redirected -- proxied path, D1-gated.
OPTIONS Unchanged preflight.

Both entry points (zarr.nemar.org host fork and the /zarrproxy path
mount) are served by the same code, redirect included.

Test plan

  • cd backend && bun test -- 1410 pass, 0 fail (includes 41 new/updated
    tests in zarr-data-cache.test.ts covering: the redirect routing fork
    at every edge case, zero D1 reads / zero upstream requests on the
    redirect path (including for a private dataset), the /zarrproxy
    mount, telemetry bytes for bounded/suffix/open-ended/missing/malformed
    ranges, the isRedirectCandidate and bytesFromRangeHeader pure
    functions, and the rate-limiter exemption against a real (non-bypassed)
    rateLimiter with a real in-memory caches.default stand-in)
  • Updated several phase-1 tests that exercised store-object paths with no
    Origin (now redirect-eligible) to carry an allowlisted Origin, or
    retargeted onto index.json where the scenario specifically needed a
    blocked Origin to still reach the proxied/cached code path
  • bun test test/zarr-data-route.test.ts from the repo root -- 10 pass
    (pure allowedOrigin/corsHeaders/cacheControlFor tests, unchanged
    by this phase)
  • tsc --noEmit (root and backend) and biome check clean on both
    changed source files
  • Five one-line mutations (redirect predicate forced false, D1 gate
    moved onto the redirect path, telemetry detail/bytes arithmetic
    broken, rate-limit exemption removed) each reverted after confirming
    the expected new tests -- and only those -- failed

Closes #1061.
Part of epic #1181.

Non-browser GET requests to a zarr store object (no allowlisted
Origin) now 302 to the public S3 object instead of proxying through
the Worker, never touching D1, the edge cache, or an upstream fetch.
index.json stays proxied/edge-cached/D1-gated, HEAD is never
redirected, and the rate limiter exempts redirect candidates from the
data-ip bucket via the same predicate serve() uses to route.

Closes #1061. Part of epic #1181.
s3PublicUrl interpolated the decoded key unescaped, so a literal '#'
truncated the redirect Location at the fragment, '?'/space broke the
request line, and a non-Latin-1 character made workerd refuse to send
the header at all (Bun tolerated it, so tests stayed green). It now
shares canonicalCacheUrl's per-segment encodeURIComponent treatment
via a new encodeS3KeyPath() helper, so the fetch target, the cache
key, and the Location all agree.

Exported s3PublicUrl for a direct unit test of its real (non-s3Base)
branch. PR #1202 review item 1.
ZARR_OBJECT_PATH_RE's \/zarr\/?(.*)$ matched "bogus" as rest for
/on000001/zarrbogus, exempting a path that never reaches serve()'s
route from the rate limiter. Anchored on \/zarr(?:\/(.*))? like
ZARR_PATH_RE, requiring end-of-string or a literal slash after zarr.

isRedirectCandidate never validated the dataset id either, so
zz000001 (wrong prefix), nm1234567 (too many digits), and nm999999
(over the 99999 cap) got redirected to a garbage S3 URL and wrote an
invalid id into Analytics Engine, while the same request with a
browser Origin correctly 404'd. The regex now captures the id and
isValidDatasetId gates it; a malformed id falls through to the normal
proxied 404 path instead of a separate D1-free branch.

Also drops the stale "catalog.json will need the same exemption"
comment: catalog.json is a separate route with no <id>/zarr/ segment
and can never match this regex at all.

PR #1202 review items 2, 3, 7 (comment only; item 7's tests follow).
"bytes=100-50" is syntactically a valid single range (parseCacheableRange
only checks digit shape, not end >= start), so it computed -49 and that
landed in the Analytics Engine data point. Real S3 would 416 an inverted
range like any other out-of-range request, so there is never a real
byte count for one; fall back to the same "unknown" 0 as every other
unmeasurable case.

PR #1202 review item 4.
Replaces the unconditional per-request console.info the zarr
redirect exemption used to emit with an observe-only mode on
rateLimiter itself (a new RateLimiterOptions.observeOnly). A
redirect candidate now counts toward the SAME IP-keyed data-ip
bucket a proxied request from that client would be enforced
against, but never blocks -- so a later proxied request from a
client that's been hammering the bucket via redirects still 429s
correctly. Logs at most one console.warn per window, the first
time the bucket crosses the point enforcement would have tripped,
with a hash of the IP (not the raw address) rather than a log line
per exempted request.

Updates AGENTS.md's zarr paragraph to match, and corrects a related
inaccuracy: catalog.json is proxied and edge-cached but has no D1
gate of its own (it lists only already-public datasets), unlike
index.json.

PR #1202 review item 5.
AccessEvent.detail's zarr contract listed only index/metadata/chunk;
the redirect branch's "chunk-redirect" marker (#1181 phase 6) was
undocumented at the source of truth for the field.

PR #1202 review item 6.
Route-level companion to the isRedirectCandidate unit tests added
earlier: a bare GET /catalog.json still gets a 200 from S3 via
serveCatalog, never a 302 -- this route has no <id>/zarr/ segment
and never consults isRedirectCandidate at all.

PR #1202 review item 7 (tests).
- /zarrproxy + allowed Origin + store object: proxied bytes with CORS
  (item 8)
- HEAD tests assert the fake upstream actually received method HEAD,
  not just a matching path/range (item 9)
- Private dataset redirect safety: follow the 302 Location for real
  and confirm S3 itself 403s an unauthenticated request under the
  private prefix (item 10)
- 8 concurrent redirect-candidate GETs: all 302, zero upstream, zero
  D1 (item 13)
- Re-verify range-key hit/miss, the full-object size cap, and the
  206-without-Content-Range fail-closed path via a no-Origin
  index.json request, since those mechanics moved to allowed-Origin
  store objects earlier in this PR (item 12)

Items 1-7 and 11 landed in earlier commits on this branch.

PR #1202 review items 8, 9, 10, 12, 13.
@neuromechanist
neuromechanist force-pushed the feature/issue-1061-phase6-redirect-contract branch from 52c2f89 to 8b35ec7 Compare September 2, 2026 18:23
@github-actions

github-actions Bot commented Sep 2, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR1202

Version: 0.9.12-PR1202.23f4b7a
Commit: 23f4b7a


🤖 Updated on each push.

@neuromechanist
neuromechanist merged commit 04ec7fa into feature/issue-1181-epic-zarr-serving Sep 2, 2026
12 checks passed
@neuromechanist
neuromechanist deleted the feature/issue-1061-phase6-redirect-contract branch September 2, 2026 18:35

This branch was previously deployed

1 inactive deployment
npm — 23f4b7a3 Deployed Sep 2, 2026 by neuromechanist via publish #2179
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant